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Foreword 



The Educational Resoiirces Information Center (ERIC) is a national 
information system developed by the U.S. Office of Education 
and now sponsored by the National Institute of Education (NIE). 
It provides ready access to descriptions of exemplary programs^ 
research and development efforts^ and related information useful 
in developing more effective educational programs. 

Through its network of specialized centers or clearinghouses^ 
eadi of which is responsible for a particular educational area^ERIC 
acquires^ evaluates^ abstracts^ and indexes current significant intov 
mation and lists this information in its reference publications. 

£RIC/RCS» the ERIC Clearinghouse on Reading and Communi- 
cation Skills^ disseminates educational information related to 
research^ ins^ction^ and personnel preparation at all levels and in 
all institutions. The scope of interest of the Clearinghouse includes 
relevant research reports^ literature reviews » curriculum guides and 
descriptions^ conference papers, project or program reviews^ and 
other P^t materials related to all aspects of reading^ English^ 
educational journalism^ and speech communication. 

The ERIC system has already made available-Hhrough the ERIC 
Document Reproduction System— much informative data. How* 
evetf if the findings of specific educational research are to be intelli- 
gible to teachersand applicable to teaching, considerable bodies of 
data must be reevaluated » focused^ translated, and molded into an 
essentially different context. Rather than resting at the point of 
making research reports readily accessible, NIE has directed the 
separate clearinghouses to work with professional organizations in 
developing infonnatbn analysis papers in specific areas within 
the scope of the clearinghouses. 

ERIC is pleased to cooperate with the National Council of 
Teachers of Ehglish in making ^4 Procedure for Writing Assessment 
and Holistic Scoring available. 

Bernard 0*Donnell 
Director, ERIC/RCS 
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Introduction 



Writing assessments can be used for a number of puiposes-scores 
for explaining needs in Title I and Title IV^ applications^ definition 
of minimum competency standards required by state law^ group 
profiles for evaluations of writing programs^ and classroom instruct 
tion. No matter what the purpose^ one of the most productive 
ways to assess writing is the holistic scoring of writing samples^ 
using an overall impression to rate a paper on a numerical scale. 

^ting is assessed in a number of other ways: (1) multiple 
choice tests» selecting the best item or the right item; (2) primary 
trait scoring of a writingsample) identifyingthe presence orabsence 
of traits required by the rhetorical situation in the writing assign- 
ment (problem-solving letters must identify the problem); (3) ana* 
lytical scoring of a writing sample^ focusing on problems which are 
not speciBc to a given writingassignment (mechanics^ word choice); 
(4) discourse scoring^ using counts of T-units^ cohesive ties» and 
other discourse units. These other ways of assessing writing have 
their U5es» but each has an important deficit. The multiple choice 
tests attempt to assess writing skills without asking students to 
write somethings and the error counts assume that the total of 
countable errors accurately reflects a paper's overall merit. 

The procedure used in this book will illustrate how to score 
papers with holistic methods and analyze the results with methods 
horn primary trait scorings analytical scorings and discoturse scoring. 
Writing assessmentsusingmultiple choice tests will not be examined. 
Holistic scoring assumes that in a writing assessment students should 
write> and that error counts alone cannot accurately reflect com* 
petency levels. Holistic scoring assumes that what is needed first 
are some prototypesof competency levels»and second»some feature 
analysis of the prototypes themselves. The method presented here 
assumes that the best way to get prototypes is for trained readers^ 
particularly experienced classroom teachers^ to read samples of 
student writing and score the samples as a whole or as a single 
impression. In other words, the whole of a piece of writing is 
greater than the sum of its parts. 

1 
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The use of prototypes to define a category of meaning has been 
investigated in a number of studies by Eleanor Rosch (1977). There 
have been basically two approaches for investigating what proce- 
dures are used in identifying meaning— the network approach and 
the prototype approach. The network approach says that people 
judge the meaning of a work by moving through a hierarchy of 
semantic levels— ftom animate (inanimate) to nonhuman (human), 
to species, to breed, and so forth* The prototype approach says 
that words have a meaning structure not captured in hierarchies, 
that, in fact, the structure is built around a range of what is typical. 
In one study, Rosch showed that when people hear the word ^^^d, 
what they have in mind is a prototypal bird that is the most typical 
in the category. She asked people to write sentences using the 
word 6ird, and then she replaced bird Ath. some member of the 
category (osfric/i, robing eagle) and asked other people to rate how 
reasonable the meanings of the new sentences were. People found, 
for instance^ that using the word c/tfc^en, certainly a member of 
tlie bird category, often produced nonsensicalsentences. Tlierefore, 
the people who used bird were not thinking of c/tic/£en as their 
prototype. Although a chicken has many of the characteristics 
of a bird— wings, beaks, feathers, and so forth^*most people recog* 
nize intuitively that dticken is not as typical of the meaning of 
bird as is robin or eagle. 

The same situation exists in holistic scoring. Even though one 
can list all of the characteristics of a good piece of writing (clarity^ 
coherence, complete sentences, smooth transitions, good spelling 
and punctuation)^ the best way to identify a good piece of writing 
is to ask people to select typical samples which they rate highly. 
This process has had a successful history in schools, districts, and 
states. Paul Diederich (1974, p. 55), who has done more than anyone 
else to develop holistic scoring procedures for use in the schools, 
describes vrhat happens to teachers who have paarticipated in holistic 
scoring for two years: 

After two years (at niost)^ they move easily and naturally into the 
of standard scores as a quicker and eajier way to indicate the!r 
judgment of the general merit of a paper. We call this ''ratings on 
general impresson,'* but it ts no longer blur: itis a quick summing 
up of charactertetics tha*^ determine whether a paper is high, 
medium, or tow in general merit. The teachersalsotiavea common 
vocabulary for discussing the merits and defects or papers on 
which their grades disagree. 
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The National Assessment of Educational Progress has used ho- 
listic scoring to measure the writing competency of three ages^9, 
13» and 17< The nine- and thirteen-year-olds wrote three essays each 
and the seventeen-year*olds mote two» and samples were gathered 
from the three age groups throughout the country* John Mellon 
(1975) in his report on National Assessment says: 

Firs( of all, teachers everywherij should applaud (lie fact that 
NAEF elected, even in lis ftrstiound writing assessnient, not (o 
employ (he muKipk^cholce questioning fomiat so widely used In 
commercial tests of writing ability. Equally enlightened was the 
recognition that the information gathered Is baseline data. No 
prior norms or absolute standards were invoked. 

This last point is especially important for schools or districts 
just beginning to gather writing samples. The first two or three 
collections provide baseline data» not absolute answers^ In later 
years, the baseline data can be used in the interpretation of results. 
Mellon^ in the same report (p, 23)> comments on the reliability of 
holistic scoring: 

liolistic scoring techniques have been extensively researched 
over the past twenty yean, parttculaily by penonnel of ETS In 
connection with essay exercises used In various College Boatd 
examinations, ft is rinown, for example, that Inter-rater reltablllty 
correiatlons (measures of the extent to which raters agree vttih 
one another on tne rating assigned to a given essay) reach as h1^ 
as .70 to .80 and above li raters are given sftcclal training sessions 
prior to their work. 

Equally high correlations are found between initial ratings and 
delayed reratingsof agiven essay by the same reader (after special 
training), thus verifying Intra-rater consistency. Put more simply, 
wc know that trained readers are consistent In thefr own o\-erall 
quality ratings and agree with the ratings of other readers about 
two-thhds of the time. This is a far higher percentage than we 
Initially thought, on the basis of earlier studies of judgments of 
writing ability, could ever be attained. 

Charles Cooper (1977> p. 19) makes the same point in his review 
of the research on holistic scoring: 

As emphatically as 1 can, let mc correct the record about the 
reliability of holistic judgments: When raters are from similar 
backgrounds and when they are training with a holistic scoring 
guide-either one they borrow or devise for themselves on the 
spot^they can achie\^ neatly perfect agreement in choosing the 
better of a pair of essays; and they can achieve scoring reliabilities 
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>n tbe high eighties and low nineties on their summed scores from 
mulUpte pieces of a student's writing. 

The experience of the National Assessment of Educational 
Progress^ the Educational Testing Gervice^ the school districts men- 
tioned in this book» and the comments of Paul Diederich^ John 
Mellon^ and Charles Cooper-all of these sources of information 
can be used» if necessary^ to convince school authorities of both 
the usefulness and reliability of holistic scoring in miting assess^ 
ment if readers are given adequate guidance and training. Wliat 
follows is a description of how to make a holistic assessment of 
student writing, focusing on (1) topics^ (2) directions, (3) scoring^ 
and (4) reports. No attempt is made here to examine all of the 
problems^ doubtful procedures^ or research issues in holistic assess- 
ment. The primary emphasis is on what works and what practical 
difficulties to avoid. The general consensus seems to be that, 
despite the many research issues and doubtful procedures, holistic 
scoring is still the best way to assess writing. 
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Topic Selection 



The topics are selected after decision-makers have generally agreed 
on the goals and objectives of the writing assessment. The three 
most common goals in writing assessment are evaluation of cate- 
gorical programs (Titte I» Title IV), determiiiation of whether indi* 
vidual students have attained minimum competent^ (graduation 
requirements mandated by state legislation), and estimation of 
instructional needs (school board or school site decisions). 

The objectives are usually defined by acommittee of community 
or professional representatives, or both. For instance, the National 
Assesi;ment of Educational Progress (NAEP), which has been 
arstgned the goal of sampling writing skills at various ages and in 
various parts of the country, appointed a seven-member panel to 
adopt objectives for a writing assessment* NAEP*s staff then 
designed topics (NIE, 1970) to meet the objectives; 

OBJECTIVE (NAEP Bpt. 3, pp, 16-17) 
Write to communicate adequately in a business or vocational 
situation (infonnatlon and application fonms, mall order letters, 
business invitations, formal letters). 



Pill in the blanks below. Do not use your own name Each boy 
should call himself Adam Baker Carson. Each girl should call her- 
self Alice Baker Carson. You may make up the rest of the infor- 



TOPIC 



matlon. 
Name: 



Middle biittal 



Last 



First 



Address; 



Number 



City or Town 



^tate 



Street 



Zip Code: 



Date of Birth: 



YelF 



Month 



Day 



Todays Date: 
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OBJECTIVE (NAEP RpL 3, pp. 12-15) 

Write to communicate adequately 1^ a social sttuatlon (letters^ 

directions, formal notes^ addressing cn\'«1opes, Invitations). 

TOPICS 

UUerOftnmtation {age 13, Tabic lfnumbcr31 
About throe months ago, Leo Logan moved from the city to 
the country. Ills father bought a farm, and now Leo's address is 
Rebel Road^ Rural DeHveiy No. 1 , Harris, Nebraska 69000. 

Leo likes the country* but he misses his old friend Ozzte Drake. 
Leo^s mother says, "Why don^tyou write to Ozzieand invile him 
to visit us for a week this summer?" 

Write Leo^s letter of invitation to Ozzie. 

[Approximately 1-1'^ pages of lined space were prodded for 
the response.] 

Recording a Telephone Message [age 13; Table 1, number ^) 
You are going to hear a telephone convetsation betui^een two 
boys, Al and Ben. During the convetsationtyou will discover that 
Ben is going to have to write a note to his mother. Listen carefully 
to find out the things that Ben will have to say in his note. 
[Children then listened to the conversation reproduced below. It 
was not printed in the children*^ booklet.] 

Al: Heyt Ben^ this ts Al. 1 called you to remind you you're 
supposed to come down to my house for supper tonight before 
the game. 

Ben: Oh yeah, I remember. Your mother and father arc going to 
pick me up on the way back from the barbershop. What time do 
you think youH get here? 
At: Oh, around 5 o'clock. 

Ben: O.K. Td better leave a note for my mother. She won*t be 
home unitt 5:30 and maybe she's forgotten that I won^t be eating 
at home. 

Al: You^d better remind her of the ball game, too. She^ supposed 
to pick us up afterwards, Isn^t she? 

Ben: ThaCs right. Your father is takit^ us to the game but my 
mother Is bringing us homa Thanks for reminding me. V\\ write 
the note right away. 

You wilt now have another chance to hear the conversation 
between Al and Ben. Listen to it carefully again and then, when 
the boys have finished talking, write the note that you think Ben 
should leave forhSs mother. 

[Approximately one^lialf page of lined space was provided for 
the response.] 

OBJECTIVE (NAEP Rpt. 3, pp. 53-64, 69*106) 

Wti te to communicate adequately In the scholastic ^tuation (notes 

and announcements, narrativ*;, descriptive, and exposttoiy essays). 
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TOPICS 

Essay Topks for Age 9 
Going to School 

Think about what happens when you go to school. Write a Mtt]i^ 
stoiy that teUs what you do from the time you leave wliete yt^u 
live until you get to school. Be sure to include eveiything that 
you think is Important 

Forest Fire 

Here is a picture of something sad that is going on in the forest. 
Look at the pict^ire for a while. Do you see the forest fire? Write 
a stoiy about what is liappenmgin the picture. This is an important 
stoiy because you v/ant people to know about this sad event 
[The picture depicts a forest fite with animaJs swimming across a 
river rapids to obtain safety.] 

Astronaut 

Hereisa pictute of an astronaut on the moon. Look at the picture^ 
for a while and think about what Is happening. Now* pn^U^nd that 
you were thi^ astronaut, and write a stoiy about your walk on the 
moon* 

Essay Tbpfcs for Age 13 
Historical Event 

If you could make an event in American history happen again so 
that yon could play a part in it, which one would you choose? 
Write a composition In which you describe the ex'cnt and explain 
why you chose that particular event and what part you would 
like to play in iL 

Famous Person 

Most of us look up to some famous person as a representative of 
the things we believe in or as the kind of person we would like to 
be^ This person may come from atiy part of our society. For 
instance, we might admire Winston Churchill or Martin Luther 
King, Walter Schirra or Mickey Mantie, Florence Nightin^e or 
Barbra Streisand. No matter where this person comes from or 
what kind of work he or she does, however, we can recognize 
such traits of greatness as determination^ physical courage, the 
ability to inspire others, and faithfulness to some worthy cause. 
Think about a famous person whom you admire. Select a particu^ 
larly admirable characteristic or quality of that person— such as 
Mickey Mantle's courage In the face of crippling physical handi. 
caps or Florence Nightingale's determination to fight against 
strong governmental p^ssure. Write an essay of about 
words describing this characteristic or quality. Be sure to provide 
an illustration of it from the person's life. Try to slK^w that the 
person is great at least partly because of this characteristic or 
quality. 
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Essay Topics for Age 1 7 
Tomato Lady 

Here is a picture of a woman with some tomatoes^ Look at the 
picture for a while and decide what is going on. When you have 
decided, wtite a story that tells what is happening tn the picture 
and what is likely to happen next. 

Famous Person 

[Topic identical to that used at age 13.] 

One must vary the topic to fit the need for different modes of 
writing. The most common modes aie description^ narration (fic- 
tional or actual)^ exposition (analytical)^ or a report (newspaper^ 
magazine documentary). Letters^ both personal and business^ and 
persuasive essays are just two of the many possible variations on 
expo^tion (editorials)^ report or exposition (business letter)^ and 
narration (personal letter). One parent committee^ convened to 
decide the objectives of a writing assessment^ called for an assess- 
ment of writing skills in letters and argument. The teacher com- 
mittee decided to measure the tetter and the argument with the 
following topic: 



Writbg Sample— Letter 

The following advertisement appeared In the local paper: 

Summer Employment 
Earn and Learn 

On*the*job ttainlng for future positions In: 
Medical Services Aide at Highland Hospital 
Hcket Taker at local theatre 
Groceiy Clerk at local supermarket 
Plumbbg Assistant toi plumt^ng company 
Assistant to the manager of a circus 

Earn $^50 or more per hour while you learn a valuable skill. 

Send letter of application to: 

Ms* Laura Jones^ Opportunity Training Center 

2212 Fhiitvale Avenue, Oakland, Califbmia 94231 

Pretend that you are applying for one of the Jobs listed. Write a 
letter explaining why you should be hired. You should give infor- 
mation about yourself. The Information may be actual facts, 
made-up Information^ or a combination of fact and fiction. The 
letter sliould help you get the job. Be convincing. your best 
work. 

Use the side of your answer sheet labeled "Writing Sample- 
Letter.** 
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This topic allows the student to use actual fects or to make 
them up^ What is troublesome about this assignment is the defini- 
tion of a business letter. Some believe that the letter should have 
no indentation for paragraphs^ some say yes. Others say the spacing 
between the date and the body of the letter is a critical feature^ 
and indeed some businesses themselves stress the importance of 
this spacing, The way to resolve the issue is either to ignore the 
finer points of business letter form or tQ convene a committee of 
business teachers to pick the anchors. The point is that some modes, 
like the business letter^ raise technicalissues which must be resolved 
before the reading begins. In any case^ the first step in selecting 
the topic is to decide the mode to be assessed and then have the 
teachers write on the possible !;opic or topics. 

Those schools and districts that conduct holistic scoring over 
several years will be able to identify topics that work. Katharine 
Blickhahn (1973) has reported that two topics were particularly 
successful in the six-year assessment of the Tamalpais School 
District: 

Spring 1970 

If you had to choose to be some fomi of life other than a human 
beingf what plant or animal fonn would you choose? In developing 
your piece of writing^ describe what it would belike to live as 
that form oflife^and explore some of the reasons foryour choice. 
You may write a letter, a stoiy^ an autobiography^ an essay or 
^y other mode. You may write from the point of view of that 
life fomr-pietend that pEant or animal has the power of thought 
and speech and use the first person. Assume you are writing for 
an audience of students. 

Spring 1972 

Write about an object you are especially attached lo^ something 
which has deep personal meaning for you^ something which has 
become part of your life. You might want to consider the way 
you discovered it^ the way it came into your life, the way it has 
taken on mining through time. You may write a journal entiy, a 
letter, a story, a bnef autobiography, an essay, or other form. 

The California writing assessment for high school seniors^ a one- 
year project, surveyed the experience of several districts and selected 
the following topics (California State Department of Education, 
1977): 

Description or Report 

Here is a diagram of an automobile accidi*nt. S>tudy (ne diagiam 
for a while and then describe the accident in your own words. 
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Exposition 

Not alUnventions have been good Cor all mankind. Nameone inven- 
tion we would be better oft without. Discuss why we would be 
better off as a civilization without that invention. 

Letter-Report 

Look at the picture carefully. Pt^tend that you know about the 
situation because you were there. Then pretend you are one of 
the following people: (1) an older student helper writing to the 
piindpal of the sdiooU (2) a parent writing to the parent of the 
little girl; or (3) a member of the audience writing to the music 
teacher. 

Then write a letter to the person named, explaining what you 
saw and what you think about it. 

Directions 

When we make or do something^ we usually follow certain pro- 
cedures. There are certain steps to follow in baking breads tarring 
a rooft cutting a pattern, paintinga house^ repairingacar, develop- 
ing film, changing a tire, and performing other such activities. 
Choose something that you know how to make or do. Describe 
from the beginning the steps that you follow in order to m^e it 
or do it. Make the directions as simple and clear as possible. Some- 
one who is not familiar with the process that you are describing 
should be able to understand and follow your directions. 

Another California writing assessment^ tliis one an evaluation of 
writing samples &om those attempting to secure a high school 
diploma by examination^ selected the following topics in 1976- 
1977 (Bernstein and Tanner^ 1977): 

November Test 

A. Write one or two paragraphs explaining why you would or 
would not report another person you saw cheating on an im* 
portant exam such as the CHSPE {CHlifomia High School 
Proficiency Examination]. 

B. Write a letter of one or two paragraphs to a friend or relative 
who is seriously considering taking the CHSPE. On the basis 
of your knowledge of the CHSPE and reasons for taking it, 
explain in a letter why you believe the persorx to <ivhom you 
are writing should or should not take the exam. 

Afarch Test 

A. Write a one or two paragraph letter of recommendation for a 
friend or relative you know well Assume this person will use 
the letter in seeking the kind of job for which he or she is 
best qualified. 

B. Write one or two paragraphs explaining why you believe physi- 
cal education should or should not be required In high school. 
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Jutta Test 

A. Write one or two paragraphs explaining why you do or do not 
believe that the Equal Rights for Women movement has been 
good for our sode^. 

B. Write a one or two paragraph letter to a friend ot relative ex- 
plaining why you decided to take the Palifomia High School 
Ptoftciency Examination. 

Julia Gottsman reports success with ^^Describe your favorite 
food" (Los Angeles County Superintendent of Schools^ 1978K 
and readings in Oaldand^ California^ have had success vrith the 
following topics: 

1. Describe one thing you would change about school and one 
thing you would not change about school Tell why you would 
change it or why you would not change it. 

2, "Clothes make the man or woman/* Explain why you agree or 
disagree with the statement. 

3< Describe two ways to save water and give examples of people 
who have done it. 

This last topic is no longer workable but, at the time » it succeeded 
because every San Francisco Bay Area family was worried about 
living within an assigned water quota. Topic two worked better in 
some schools when the last sentence was changed to read» "Explain 
why you agree or disagree with people who believe this statement/* 
The change provides a different meaning. But changes in meaning 
do not matter. One topic is better than another because students 
find they can readily vnrite on it in the desired mode. 

Topics can vary in the amount of planning help that they give 
students. The Spring 1972 topic ftom Tamalpais^ mentioned earlier^ 
provides some suggested outlines for the content of the topic» but 
the Spring 1970 topic provides suggested points of view and dif fe^ 
ent modes without much help with specific content. The letter* 
report &om the California assessment^ included above» gives 
point-of-view choices but it does not suggest a content outline. 

Topics can also shift the mode from report to analysis by using 
words tike describe and why (CaUfomia assessment^ **Desa1ption 
or Report** and **Exposition/' respectively). The choice of words 
can be critical. For instance^ some students thought **describe the 
accident** meant that one should tel! about the sounds of the crash 
and the dents in the cars. For these students^ ^^Report on how the 
accident happened** might have been clearer. 
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Topics are always threatened with the problem of being non- 
functional. For some students, if they cannot at least imagine 
some functional use for the writing, they cannot do well in writing 
prompts. The prompt on "Directions*' from the California assess* 
ment is an example of such a situation. A student may know how 
to give directions for all of the things listed^ but the nonfunctional 
situation keeps the writing fromeither beginning or being effective. 
A more functional topic might be ''Give directions for getting 
from your house to your present location.** 

The same topic raises problems of organization. Many young 
writers will write their directions in a list: 1, 2, and so on. This is 
a perfectly reasonable method of organizing directions, and the 
raters of the writing samples will find very little relevant diversity 
in method of organization^ If organizational skills are to be con* 
sidered in the assessment, then another topic might be used. 



Hie Function of the Prompt 

A topic should be a prompt^ a way to stimulate fluent student 
writing, not a test of a student*s knowledge of special facts. The 
prompt should enable the students to start writing quickly, provid* 
ing no obstacles of subject matter. The prompt should also be 
direct and easily understood. Sometimes teachers grow weary of a 
useful prompt, and » believing ''it *s time for a more creative subject/* 
they turn to such misguided efforts as the following: 



NO MATTER WHERE YOU 
GO, THERE YOU ARE. 

EXPLORE INNER SPACE. 

YOU ARE HERE, 
WHERE IS THAT? 
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Hunk about the picture and the words. Write a prose essay about 
the relationship between th€ picture^ the words^ and you. Re- 
member to write prose— no short stories^ poems^ or pl^s. 

The student responses this prompt were usually a struggle to 
come to grips with some central idea or theme. Examples (with 
transcripts) are given on pages 14-18. 

If the teachracs who gave the previous prompt had written on it, 
they would have found how difficulty if not impoBsible^ the prompt 
was. A prompt should always be field tested by having teachers 
and a dozen or so students write on it. 

The field testing will usually identify the major problems. The 
following topic was eliminated after the teachers found that the 
requirement of special knowledge became an obstacle to writing: 
"Ebcplain your agreement or disagreement with the following state* 
ment: The history of women in our country is the hjlstory of 
oppression.'* 

Two teachers admitted that they did not know enough about 
the history of women to write a fluent essay on the subject. This 
particular problem is esprcially relevant in the prompt lelected for 
exposition. After all^ how is one to argue one side of tn issue if 
one does not have special facts distinguishing the various sides of 
an issue? 

Another problem in prompts is the open-ended fonn. These 
prompts allow the student to write in any form, firom penonal 
notes and journal entries to formal exposition. The readers of these 
writing samples cannot judge the qualify of the writing because 
the student writers can claim they are writing personal notes to 
themselves, a rhetorical situation not demanding the usual conven- 
tions. T!he same problem Is produced by allowing student writers 
to fantasize any rhetorical situation. 

The grade level is another consideration in the selection of a 
prompt. In an Oakland^ CaUfomia» reading forfourth and twelfth 
graders^ for instance^ the teachers selected as a topic ^'Describe 
your favorite person.** In the third and fourth grades, the topic 
had to be changed to ^'T^U a story about a favorite person** because 
many students in these grades often interpreted "describe** literally^ 
as in ''He is six feet tall. He has blue eyes and brown hair.** 

Kenneth Williams (1977) reports a similar experiencein Modesto^ 
California. The fourth grade teachers reported that some topics 
worked better when "tell about** was substituted for ''describe.** 
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^(MOUiMM^Mi^^^ ^^^^^^^^ 
(ftc^t^ ^£t^^ -'^<^^ 

^ OcJi^^ /ULAJ^'^ ^ ^ ^^^^^ 

Hi/f^- J /m^ 4cU^ -oM^ ^ i<ja^ 



[Transcript] Here I am, just a person in a place that means little to my 
future. I stare into the distance serching the vnldersness for scrvJval and 
begin reaching out into outer space for help. My nouth hangs open in hopes 
that what is in store for me finds me well 

Tm here in highschool, but for how much longer can I hold on to my 
friends, teachers and belongings. There is such a vast future ahead of me and 
Tm bewildered at the sight of new adventure. Ves, in September I must start 
all over new, making sure the goals my parents have led me to are conquered. 
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X oun vwiJ ^ bmjt' J 5f lCL oUm^ itnoco uJ^ot 
J a^m . ^ Ufcc 4o Spoof pcjojoU^itj '*^9' 

-^'ofc^ ±h\vii^5 ^o^tlt^.l ^i>QUA^e' XOe £^of a. 
realXcj cto vru/y\cP doiioo {jitho±^ re<(ijjr^ of 

(j^crcoi^ ^^yju^tjc ojo\x: V^'i^rtxjh, >aii c^uji^-^ uf€ <xii 
(^Ko tl^s Olt^^ 07ul$ ijj-ho ynahe 

J: cto tf cvocu>(. a +^ no u^^ 4o i-n^ Vo 



and that they have raised me well enough to make them proud of me. I ha\G 
to make sure that my study habits are good itnd tnat my personality catches 
lots of friends. Tin on my own now with little home security and no more 
direct orders from home base. 

It's sort of scarey really, and when you come right down to it* realizing 
that soon you have to go up to bat "on your own it becomes a dark and 
lonely worM. Tm no longer kept by school* parents or society, but by my 
own. J must seek what 1 want and how 
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Co^^Xd K<<^ u^ni^io^ |b^u>t<-« 



[Transcript] English D^pt. Writing Exam/I »m me, tnit I stiU don't know 
wiiat I am. I like to spoof people by playing games, maybe thtts me. I don*t 
think so thougii. Deep inside I know I want to do something great to benefit 
everyone. Maybe my time will come, who^'s to say? Right now I'm happy, but 
I am happy most of the time. My moms says that I don^t complain much, 
thats because I want to learn to take things easily. I figure Pve got & long time 
to do, and get done. Mavbe I really do mind doing what is required of me to 
do, I just don't complain about It. I like to do extra little things to make 
those I love, happy. It gives mea warm, tingly feeling inside, especially .irh.^n 
I get a reaction. Tve always wanted to go back in time to the way it use to be. 
I know it would never work though, because we all have to many hostilities 
towards things to make the world any better. 1 feel that descrimination is of 
the mind, and those who feel this are the ones who make these things they 
dcscrjminate against look bad. I even have trouble figuring out why I*m here* 
trecause when I do^ what society has brought me up to think are naughty 
things, I feel very tradly, or I do it again. Its no use to try to figure myself 
out, because I never want to grow up I know 111 have to begin to accept 
more and more responsibilities as I get older but in a sence^ I never will 
grow up, I know! Tve always wanted to live a perfect tife. especially when 
I create a family, but what Is a perfect life? Should I live like my parents 
iike the bible, like the poor, like the rich, like the foreign? I guess I'm just 
going to wait, and live like me, and live every day the way it comes. 

(I could keep writing for days, but we're not suppose to. I wouid probably 
even contradict myself!) 
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•r^t'S fuc^i^/h^ See's -^Ae t^a^M 

pep /-^e^ / ^e^^jj X f^i/ 
Soh^&<^ha / pu'^zje^f J, y ike 

OJ- CO Me ffoi^ ^ A^7ai-A /ttac/^j 
Mev lAjere yffe-^Z/ sec'^.^ />^c 

So ca//ec/ ^ijLes^/o^ /■/ c^aj- 

/*u C^oj- S excuse »9t> 

a+ fic/s or Ah^^/j/a^o -/Ai?/^ 
fti^ au>/- a/of cy ^uor X 
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[Transcript] I fe«l wtio ever wrote this <luestioii see*s the world as a big bta 
In other word's Department head's, I feel somewhat puziled by the question. 
Now if it would of come from a math teacher I would of felt like they were 
Realy seeing the world this way the way of a bla. But for some strange and 
unkown reason [ feel that the person who wrote the so called question if it 
was and inglish teache"* had to be depressed about some thing {n the world. 
Because my feeling toward you, the english department is 75% of the time 
you live, {s living in happiness and fun the other 25% is really not bad at all. 
Bcatise ^1 you do is set back behind a desk belowing at kids or laughing at 
them Sometime's you correct paper's* but you redly don't have to put 
out alot of work. 

So Mr. White and the rest of the departments I have come to the con* 
elusion that the 1970 co;nposition question was terriable. I feel that you the 
department were quite dvmp to over look the fact that this was not a ques- 
tion it IS some depressed teacher's feeling's toward the world. Soiy I have to 
quit but time requlrees me to do so. 
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In the fourth grade samples below, the middle and high papers 
seem to have more fluency because the writers have decided to 
**tell about*' rather than describe. The topic vras to describe a 
favorite pet: 



I wish I had a ciocodUe a man eater U h log and fat. It wad a 
hundred pose and egle. It runs slow. 



I wish I had a brown OufCy baby monkey to play with. I would 
play with it eveiy day and feed It I would never sale it. It would 
have and soft Huffy bed »nd tt would have a Huffy, fat soft 
pillo. ! like him a loL^^e End 



If I could have anything I would want a clear blue icc cold 
swimming pool in the si^mmer. It would be so clean that you 
could see the bottom. 

In the winter I would want to own a ski resort and go skiing 
eveiy day. The snow would be so white that you would feel like 
you were skiing in whipped cream. 

I would swim all day in the summer^ and ski all day in the 
winter. life would be boiing inbetween. 

Sometimes a district or school desiring to score samples firom 
the fall and the spring, vriU need to alternate comparable topics so 
that students will not be writing on the same topic twice* Rae Jean 
Popham (1978)^ in a paper for the UCLA Wdting Project, recom- 
mended that the students bedividedintotwogroupsand two topics 
be alternated between them: 

Group A Group B 

Fall: Describe your favorite Fall: Describe your faTOiite 



During the scoring, all of the samples are put in the same stack. 
The topics do not indicate when a paper waswritten. Furthermore, 
by the end of the anchoring period, the topics tend to blend, and 
differences in what is described-obj^ct, person^ place, entertain- 
mentj gift--tio not make any significant difference in the scoring. 

Summary Checklist 

1* Organize a committee to adopt assessment objectives. 



Low 



Middk 



High 



object. 

Siting: Describe your [im>Hle 
person. 



person. 

Spring: Describe your favorite 
object. 



20 Topic Sctcction 

2. Match the topic to objectives^ selecting mode and rhetorical 
situation. 

3, Field test the topic^ having students and teachers write on it, 

4. Review the topic for problems of unfocused cieativityt 
special knowledge^ open-ended form^ and grade level differ- 
ences. 

5, Never give a prompt that can be answered with yes or no. 



Directions 



Once the topic has been selected by field testing with teachers wd 
students* many parts of the directions will have been decided. One 
obvious decision is time^ and because most secondary schools have 
fifty-minute periods* the time isueually thirty to thirty-tive minutes 
with fifteen to twenty minutes for giving directions* distributing 
necessary materials* and collecting samples. 
The typical format for directions to teachers is as follows; 

1. The students wiU vnrite on the assignment prompt in their 
En^ish classes* 

Note: Sometimes students take more than one English class. 
Teachers should check to make certain that students vnrite 
only once during the school day. Send students who have 
already vnritten on the prompt to the department chair» who 
vidll use them for collecting writing samples from individual 
teachers* 

2. Periods one* two* and three will write on one prompt* and 
periods four* five* and six will vnrite on another prompt, follow* 
ing the pattern su^es^d by Popham (1978* p. 21). In this 
way* students will not pass along the topic during lunch time. 

3. The prompt will be vnritten on the blackboard and read aloud 
by the teacher. All teachers should give students the same 
directions. (Some districts have tape*recorded teacher direc- 
tions ^d replay them so that every student gets exactly the 
same words.) 

4. Each student will be given a 3" X 5" card and asked to put on 
the card: (1) name (last name first)* (2) giade^ (3) teacher* 
(4) period* (5) first or second semester* (6) sex (male or fe* 
male)* (7) course title* and (8) code numba^ on paper. 
Note: The information requested on the 3" X 5' card matches 
what is requested in the coding for each individual psiper* 
except for the course title. See student directions for coding. 
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5* A student will collect the 3" X S" cards» and the teacher or a 
student will record this information on the attached master 
sheets: 

Student Name Grade Teacher Petiod Semester Sex Code 
BUI Jones H Ms. Smith 2 1 M 4332^ 

2a*2 

6. Each teacher will be as^gned a number^ noted on the attached 
master list of teachers in the department or grade leveL Put 
the teacher number on the board* Notice that all numbers 
have two digits* (Ms. Smith's number is 050 

7* Hand out to the student the student directions sheet. Read it 
aloud to the students* 

8* Ask students to write on one side of the paper only. If a stu- 
dent needs a second page^ ask the student to write on a 
separate sheet of paper and put th^ student code in the upper 
right-hand comer. Staple multiple sbeets together. 

9. Students wilt be allowed to use the dictionary* 

10. Teachers will be allowed to answer questions about the 
prompt^ coding papers^ and oth^ matters not related to how 
to write the assignment. Clarifying the prompt is approved. 
(Students will not he penalized for writing ofE the topic if 
the teachers hfwe decided to focus exclusively on writing* 
However^ if the teachers have decided to include reading as 
part of the assessment^ students will be penalized for not 
writing on the topic*) 

11. Teachers should make certain that they have staplers^ lined 
pap^f and No* 2 pencils. Students must write with No. 2 
pencils or dark black ink. 

12. Collect the papers at the end of the period and turn them in 

to room » complete with 3" x S'' cards and master sheets. 

Era^e the prompt if you leave the room for a period. 

A typical format for directions to students is as folbws; these 
directions are duplicated and each student gets a copy: 

1* Prepare your name code» completing the directions below. 
NAME: List the last four numbers of your phone number: 
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Not^i If you do not have a phone» put your house or apart* 
ment number^ and two or three different numbers from the 
following: 4* 5, 6, 7, 8» 9. Complete all four spaces. 
2. GRADE; Put the two numbers that represent your grade: 

Grade; 1234 567 89 10 11 12 
Nuj»ber; 01 02 03 04 05 06 07 08 09 10 11 12 

3* Teacher: Your teacher will tell you his/her teacher number: 

4. Period: Put the number of the periods 1,2^>4»5>6: 

5. Semester: If this is first semester* put a i ; if second semester^ 
put a 2: 

6* Sex: If you are male* put a2;if youarefemale* put a i: 

7. SUMMARY: Now list the numbers shown in the margin* 
putting the bottom number in the last space to 
the right: 



Notice that you have eleven digits in your number. 

8. Fill out the 3" X 5" card. Put your name (last name first)» 
gtade^ teacher» period* first or second semester^ sex (male or 
femaie)t course titie* and name code above* 

9. Put your code (see 7 above) in the upper right4iand comer of 
your paper. This should be the same number as the one you 
put on your 3" X 5" card. 

10« Write with a No. 2 pencil and keep some margins on both 
^es. You may erase mistakes. 

The information in the previous instructions vrilt have to be 
revised to ftt local circumstances. For instance* some schools have 
the quarter system^ not the semester system. Another change might 
be the need for additional information. A department might want 
to save papers &om year to year* and this would require adding a 
two-digit space so that the last two numbers of the year could be 
listed* One department added a code on student attitude* asking 
the student to list a number hdicating how much the student liked 
to vmte: U I hate to write; 2, I do not like to write; 3, I like to 
write now and then; < I like to write very much* 
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This code enabled the department to compare scores on the 
miting sample and student attitudes toward \viitine. Another 
department asked the students to rank their own papers on a scale 
of one to six» from low to high» using the same scale the teachers 
would later use. This enabled the department to compare teacher 
and student ratings. Another district asked students to rank their 
papers the way they thought their teachers would rank the papers. 
This number enabled the district to estimate how well the students 
understood the way their teachers evaluated writing. 

These directions for teachers and students assume that anumber 
of decisions have been made: (1) How long will the test period be? 
(2) What classes will be given the test? (3) Will students be allowed 
to use the dictionary? (4) Will teachers be allowed to clarify direc- 
tions? (5) WUl teachers be allowed to encourage a student to re- 
spond in whatever way possible? (6) How many numbers will be 
in the code? (7) Will there be alternate topics for pre- and post- 
testing? (8) Will there be topic changes after a break for lunch? 

Changes in grade level or objectives often change the directions 
for a writing sample. For instance^ students in the elementary 
grades are often told the topic a few days before the writing so 
that they can begin to think about the topic. Many students in the 
upper elementary grades cannot focus their attention on writing 
for nfty minutes, and a longer preparation period is needed to 
ensure an adequate sampling of elementary writing. Because 
elementary classes are not broken into periods^ and because ele- 
mentary students have a more limited attention span» the time limit 
can be removed in the elementary grades. Sometimes the objectives 
change the directions. For instance^ if the program puts a high 
priority on memorized spelling words, then the direction may for- 
bid students to use a dictionary. 

The following directions &om the San Mateo Elementary School 
District (Graham^ 1979) show the changes required because of 
changes in grade level or objective: 

Infommg Students 

A day or so prior to taking the Writing Sample^ students wiU be 
informed about the entire Wnting Sample process, l^achers will 
use this time to answer questions, establish expectations and make 
students feel comfortable with the Wnting Sample. At this time 
you may post tlie topic: "My Favorite Object" and advise the 
students to be thinking about possible subjects for their composi- 
tion. 
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PrewriUng/Brainstomting 

On Wednesday, October 10th, teachen wUl spend a class period 
involving students In piewtiting or btainstorming activities on tiie 
topic This preparation time is very important end shoutd not be 
omitted under any circumstances. 

During tills discussion period, brainstorming by students may be 
in large or small groups and witii teaciier direction siiould elicit 
ideas relative to tiie composition topic 

No Time Limit 

Altiiough the entire process should not take more than Hfty 
minutes tor the average student, you may allow slower students 
additional time (within reason) to finish their Writing Sample. 

Distribution of Materials 

Approximately five minutes should be allowed for the distribu- 
tion of paper. Students are to use No. 2 pencUs only during the 
Writing Sample. The handwriting must be dark enough for xerox- 
ing. This time should also be taken to make sune students'' papers 
are properly coded Please remind students using more than one 
page to code each page accordingly and staple pages together 
securely when they are finished. 

Reading the Prompt 

When six students are ready to begin, the teacher wilt read the 
following prompt to the class: 

Describe an object (not a person^ an animal, or an event) 
you are especially attached to, and tell why you feel strongly 
about it You might want to consider the way you discovered 
it^ the way it came into your life; the way it has taken on 
meaning through time (For fourth grade students, the assign- 
ment may be modified to include writing about a favorite pet) 

Testing Conditions 

It maybenecessaiy to spend some brief time answering questions 
before the writing actually begins; however, students may not 
receive assistance from any sou/re^ /.e.» teacher^ dictionary^ e/c, 
at any time during the writing or proofreading of the Writing 
Sample. 

Proofreading 

Students should be reminded to title thetrcompositionsand spend 
some time proofreading thehr papeis before handing them in. 

Special Handling for Learning Disabled (LD.) Students 
Alt LD. students will participate in the Writing Sample except 
those identined by the school LIXteacherasunableto participate. 
Please be sure to pull all LIX students* papeis and retum them to 
the building 1^.0. teacher as these papen will be scored separately. 
The scores of L.D. students will not be induded in the overall 
competency results. 
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Note that San Mateo removes the time limits but does not allow 
students to use a dictionaiy. San Mateo also announces the topic a 
day or two before the prompt is assigned and provides time for 
prewriting activities. Note also that special provisions are made for 
students who are classified 

In districts where a central district reading is not possible^ the 
distribution of sample anchors to the school and the collection of 
a school^s samples in each scoring category will enable the district 
office to have some information on how papers axe being scored. 
A central scoring is the most reliable procedure^ but the involve- 
ment of many teachers at the school site also has its inservice 
benefits. 

The most reliable scoring of writing samples takes place when 
the readers are trained together and read together in the same 
room under common direction. But sometimes districts cannot 
fmd the time or money to bring teachers together for a district 
scoring of papers. In such instances the department or grade level 
can usually arrange to score at least a sample of the papers written. 
Also» in such cases the district's language arts coordinator needs to 
provide some common anchor papers for all schools and some 
training for at least one teacher in each school. (The importance of 
anchors is discussed in detail in the section on scoring.) 

The following is an example of directions to teachers when a 
district office scoring cannot be aitanged : 

To: Leaders of readings at school sites 
From: Language aits coordinator 

L Work closely with the chair or trained person in your department 

or gmde level. 
2. The district prompt and directions are enclosed. 
3w During the scoring, prepare a desrription of the scoring categories. 

Please send this to the district office. 

4. IVy to give the writing assessment in your classes during the week 
of October 4-8. 

5. Enclosed are some recommended anchor papers for the prompt 
selected. Use these to practice anchoring the scoring categories. 
Send to the district office (with the scoring descriptions) examples 
of the scoring categories in your school. 

6. The scoring room should be a larg^ and pleasant place. 

One set of problems in the direct./)::S for a writing sample focuses 
on the reproduction of papers for anchoring^ analysis^ and reports. 
Students should use No. 2 pencils^ should keep at least small margins 
on both sides^ and should keep some space at the top and bottom. 
Students should be encouraged to write heavily. Light pencil marks 
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and crowded pages make dear reproduction difficult if not impos- 
sible. Furthermore) light copies make the reader's job difficult. 

Another set of problems in the directions focuses on how much 
direction to give the students. The best rule is to give as few direc- 
tions as possible and keep the prompt as simple as possible. The 
primary purpose is to elicit some quality writing &om the student. 
Other puiposes are secondary. The more students have to be told 
about the prompt in order to write^ the greater the number of 
students who will write little or nothing. The directions sometimes 
become very complex because large amounts of information are 
being coded in the numbers in the upper right-hand comer. Some 
schools have solved this problem by doing the coding one day and 
the prompt the next. The coded but blank papers are simply 
returned to the student the next day and the prompt is then given. 

A final set of problems focuses on eliciting writing horn K-3 
students. These students should be encouraged to draw and then 
dictate or write. One value of this kind of sample is that one can 
demonstrate with concrete c>xamples the importonce of drawing in 
writing development. Donald Graves (1975) has argued that draw- 
ing is a form of prewriting for young children. The two examples 
on pages 28 and 29^ ono drawing plus dictation and the other 
drawing plus writing^ were collected in an assessment of early 
elementary writing and scored by K-3 teachers. Notice that the 
child who can write ("I have a friend named Daniel'*) is also the 
one who tells his story more effectively in drawing* 

Summary Checklist 

1. Arrange the time to fit the class schedule and the attention 
span of the students* 

2 . Prepare the number code for information desired on samples. 

3. Prepare directions for teachers^ stressing the importance of 
giving encouragement but no assistance with the content of 
the prompt. 

4. Prepare coding directions for students. 

5. Mstribute to teachers the prompt^ which they will write clearly 
on the bbckboard. 

6. Stress to students the importance of dark copies and some 
kind of response. 

7. Vary directions to fit the needs of different grade levels* 
allowing K-3 students to draw and dictate or draw and write. 
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[Ttanscriptl Me and my biether have a gun. We shoct each othei with 
pietend builet. I hide my gun fiom my friend because he bieaks my toys. 
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Scoring 



After the topic has been selected* directions and prompts have 
been distributed^ and students have done the writing* all of the 
samples are collected in one room and a few teachers are selected 
to help prepare the scoring criteria. These teachers will be the table 
leaders at the district reading, each table leader having six to eight 
teacher readers. If possible* these table leaders should be experi- 
enced readers. 

The second important decision concerns how much time the 
district needs for a reading. The following formula provides a 
reasonably accurate estimate: 



Total papers X 2 (readings) X 

0-20 (third reading) 
/divided by 60 (minutes) 
X divided by niimber of teachers 

+ 2 hours/day for anchoring 



Y 
X 

' total time 
needed 



Example; 

1^000 papers X 2 (readings) = 2»000 

X 0.20 (third reading) 
2^400 divided by 60 (minutes) 
40 hours divided by 5 teachers 



2 hours of anchoring + 4 hours 
2 hours of anchoring + 4 hours 



2^400 
~ 40 hours 
= 8 hours 

+ anchoring 
= 6 hours 

(first day) 
= 6 hours 
(second day) 

Total: two days* 8 hours of scorings 4 hours of anchoring 
Note: After two days* anchoring can be reduced to VA hours. 

The third important decision is \fhether the table leaders will 
prepare a list of criteria before or after selecting prototypes from 
the samples^ and whether the teachers at the district reading will 
be given a rubric (a list of criteria) before the teachers are asked to 
score selected anchors. If one is working with experienced readers, 
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rubrics can come first or later* The process works wpU either way. 
However^ if one is \iforking with inexperienced reaoers^ the best 
procedure is to pick anchors or prototypes first and let the rubric 
or list of features evolve ftom thediscussionof the reasoning behind 
the scores. 

The scoring of the papers has two stages: (1) the selection of 
the anchors by the table leaders^ and (Z) the reading. 

Selection of Anchors by the Table Leaders 

The selection of anchois requires some decisions on number of 
categories^ the range within a category^ the effect of form and 
directions on categories^ the middle decision^ and the special prob- 
lem papers^ The most common scoring categories are four, six» and 
eight* Sometimes readers use nine> seven, ^nd Tive^ but choosing 
an odd number is always a mistake because readers wUl be drawn 
to the middle number as a compromise between high and low. The 
decision about whether to use four or eight is determined by the 
purpose of the reading. If the reading is for a general profile of 
student writing^ for only three grades* use four. Four categories 
are easier to score than eight-after all^ four categories are easier to 
rememberHbut using four makes fewer distinctions. Different levels 
of skill will be clumped together. If one intends to use the reading to 
measure change between September and May» use eight categories. 
Six is a reasonable compromise between scoring consistency and 
the various purposes of holistic scores. Six scoring categories are 
essential for more than three or four grade levels because one needs 
the spread to get the grade level effect— largely matters of phrasing 
and maturity of content. 

In readings to determine whether students have attained a mini- 
mum level of competency for high school graduation » use four 
categories if only three grade levels are involved and six categories 
if more than three. With six categories^ students at the bottom are 
more likely to show some improvement in subsequent writi g 
samples even though they still may not achieve the minimum level 
of competency. Some improvement is encouraging to both students 
and teachers. 

No matter how many categerie;; of skill are selected for scoring^ 
there will be slippage within a given category. That is> althougli 
several papers might have one number^ they often will have recog- 
nizable differences. The following papers were all put in the one» 
or low J category on a six-point scale. Theirdifferences are apparent. 
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My Nanna gave me a two dollar bill and 1 have 10 dollar atid { put 
them in my mirror. 
^*My dog" 

I have a dog. He is live in a box. Or day He playing in the yard. 
Ao July the 3 1 found some money the anout 20 dolor the thing i 
bouth a football and we aply ball evey day Itset the rain days 
and one day it got a hole In it and I pach It up and cvey ^ent then 
om air have mo air come out 

Miriam Ylvisaker (Myers and Ylvisaker, 1978)» in a commentary 
on these three papers, observes that the first is by a fourth grader, 
the second by a fifth grader, and the third by a sixth grader, showing 
that poor writers can become more fluent as they get older. Even 
tliough the third paper is much longer than the other two, it is 
still in category one. 

After deciding the number of scoring categories, the head reader 
must decide whether the samples collected have the desired norma* 
live distribution. The exact location of a paper-ihe definition of 
the boundaries of a scoring categoryHs relative. That is, the deci*^ 
sion is based on the distribution of the papers in the group. The 
number of writing samples can have, then, a distorting influence 
on the anchors selected . The larger the number of samples, the less 
likely the categories will be skewed in one direction or another, 

A good example of this problem occurs in the readings to deter- 
mine whether students have attained a minimum competency level 
for graduation. In one district, all of the 8,000 ninth and tenth 
graders wrote two samples, and these 16,000 samples were used to 
determine the range of competency within the population. Using 
this range and sample papers at each scoring level, a district 
committee decided what scoring level represented minimum com* 
petency. During the summer, the students who had failed to attain 
minimum competency in the first test took writing courses and 
then wrote a second set of samples tot scoring. This second set 
of samples did not have the distribution used to establish district 
norms because all these samples came from writers who had failed 
the first test. 

How then does one find anchor papers with which to score the 
second set of samples? The solution is to use anchor papers from 
the fu^t reading to establish the scoring categories for the second 
reading. The head reader must decide whether table leaders will 
find anchors among sampler collected or use anchors firom a pre- 
vious reading. If anchors are used from a previous reading, then 
table leaders axe asked to find other examples from the samples 
collected. 
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If anchors are to be found among the samples collected, then 
the head reader asks the table leaders to read for one hour, selecMng 
papers at random from the stack, and to identify two papers that 
could be used as anchors or prototypes for each scoring category. 
If there are six categories, then each table leader will select twelve 
papers. A paper is called an anchor because it defines a scoring 
category and keeps a reading anchored or tied to a normative scale, 
the bottom score being for bad papers and the top score being for 
good papers. 

After selecting two anchor papers for each scoring category, the 
table leaders will vmie down the characteristics which they believe 
differentiate among the categories. In a Huntington Beach, Caii^ 
fomia. Unified School District (1977) assessment, the prompt 
specified particular features: 

Please read and think about the following: 
AU of us at one time or another have feJt Jike we would Jiko to be 
something or someone else. You ha\^ a weekend io be anything 
you Jike. You may want to be a sea shell, a mo\1e star, a sports 
hero, a feather, or anything. 

Writing Assignment: 

TelJ what your now seJf will look Jike 

Explain what your new self will do 

Explain why you want to be that thing or person 

The selection of categories in the Huntington Beach reading had 
to take into consideration the three specifics in the prompt: 

A 6 response addresses all three aspe<:ts of the assignment. Though 
it may have minor faults^ it will be well organized, detaiJedi and 
well written. A 5 response includes all three aspects of the assign- 
ment but may emphasize only two. Although the 5 response will 
be less fully developed, less well organized, and perhaps less 
fluent than the 6 response, k will show an overall competence 
in writing. A 4 response includes all three aspects of the assign- 
ment, but IS weaL In the development of two. Although it is less 
fluent than a $ or 6 paper, it does display overall competence. 

Dturing the anchoring session with table leaders, the head reader 
will duplicate copies of the anchors selected by the table leaders, 
listing on a separate sheet the scores suggested i^y the table leaders 
but removing all scores from the duplicated copies. The head reader 
win then ask the table leaders to read and score the duplicated 
copies and arrange the anchors in a sequence on the table. This 
process will cause the table leaders to make a few adjustments in 
the way they scored their original anchors. The head reader will 
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then ask the table leaders to vote on the scoring category for the 
anchor papers. Those papers on which there is quick agreement for 
a particular score become the anchor papers. Those papers on which 
there is not quick ^eement are often used to illustrate particular 
problems in scoring. For instance^ in the bottom levels^ fluency 
usually can increase a paper^s score^ but in the upper levels^ fluency 
does not usuaUy increase scores. However^ a few table leaders may 
want to give fluency more weight in the upper scores* and the 
papers that reveal this difference of opinion wUl be used to identify 
the problem and reach an ultimate decision on when fluency will 
and will not increase a score. 

One of the hardest decisions for the table leaders is the selection 
of the anchors which define the difference between upper-half and 
lower-half papers. On a six-point scale* this selection would be the 
3 and 4 papers. Examine the papers on the following pages. The 
1 paper ("I was in inpression Buy my mother . - .'*) and the 2 
paper ("The person 1 remember . - .**) are shorter than the 5 paper 

("Mrs was my teacher - ^ and the 6 paper ("The person in 

my life who made a great impression on M-ne ^ . The bottom 
papers also have more mechanical errors* with the run*on sentences 
creating comprehension problems for the reader ^d the spelling 
errors producing some hesitations for the reader but few funda- 
mental confusions. Examine the four papers on pages 35-37 and 
try to guess the qualities of those in categories three and four. 

These anchors illustrate the problems of setting boundaries for 
cat^ories. The 3 wiU probably be a longer 2* and the 4 will be a 
5 with mistakes. The 4 in most readings does not have any serious 
errors* and the 3 always shows some indication of fluency. The 6* 
of course* does not have to be a perfect paper. The 6 here has 
several mistakes, one of them being a spelling error (vegtibte) and 
another being the need for a period (following die in the last para^ 
graph). Wliat tiie table leaders found to be the offsetting strengths 
'.*^3re the paper*s elegant phrasing T'-We survived the surgery^ the 
strong willed man he u^cs.'*) and telling details (''sipping cocoa''* 
and "which is now just weeds**). 

After the table leaders have read the duplicated copies and ar- 
rived at a consensus on how to score them* the head reader should 
ask the table leaders to read foranother hour* attempting to classify 
randomly selected papers in one category or another. Any papers 
judged difficult to categorize should be given to the head reader 
for duplication and later scoring by alj the table leaders- 
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[Thmscript] I was in inpression Buy my Mother she is a great women I live 
with here it is just me and my Mother My father Dont live with us me and my 
Mother go a lot off place together she gets involve with my <portand help me 
she makes sher I go to a good school and she make sher I take goo caie of my 
body and I haw and inpression on my Mother Becuse she is a great Mother 
Ms Brooks. 



^ JfvjJLLiL^ vtwv coXc^ -^A^. 



^Transcript} The person I remember the best was m> bcstfnend in elemen. 
taiy school she also live up the street from me. She had just move from 
DanvilL Her dog got lose and I helped her catch him. 

I new her for along time until she move We still try to keep in touch 
But 1 will never foreget the mernories. 
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mrfJ^/Kviressu.v^cTlVc. c5 re»^;mbur her 
^o-*jLM .bJou^e of and uindfirstiindfvq 

oh: «o!v€ mc^ht WQ^» itWo^ -tlxre. w hen Xtv 
needed V^r andt eba# ti> ■ih6 d^:^^ 
Ue. httd cur 005 and dMiS ^J^VT"^ i 
flopU dLo Wtink o/tLftu^ Yr^o^fMufk to Sflvc 

oot v>Kx Oyu^ , ,W 5^1^ J^c^^ 

^ as tlac/«rs ylp<<^^ ^ ^ 
(OuKujS vwoat UAc a dai^HtJd 

-tVCvv- C\oix -^=^^0^ ^^-M)^' ^ S^'W^ 



[Transcnpt) "Great Impression "/Mrs. Christine Wilson was my teacher 
during my elemenUiy years. She made a great Impression on me. I remem* 
ber her mostly, because of the love and understanding she gave me. She 
was always there when I needed her^ and she stjU b to thb day. We had 
our ups and downs as normal people do^ but we always managef* to solve 
our problems. 

She treated the other students as adults^ not like fourth, fifth, and sixth 
graders as we virere. Alt of the other students as weQ as teachers respected her. 
She has always made me fee] like a daughter to her 

Today we are still very close fdends even closer than before. She is one 
heck of a lady. I will always love Mrs. Wilson. 



[Transcript] The person jn my life who made a great Impression on one was 
a friendly^ strongheaded man. Mr En^trom, who lived next door to me. He 
was a man, rather old^ about 85 or so. He was a grandfather figure to me. I 
called him Grandpa Engstrom 

As a child of about 7 years, I remember running over to visit him everyday 
after school. We used to sit by the fire sipping hot cocoa and heM teU me all 
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aisaitS^ el Jiff, a ^layui^ai/ifi^ 

do. a chM aUJ 7 

,ocJu$rt Uli J> ^ 4^ jtAe, A/t/ ^^^f^ 

Mt CSCSii cutici Mcf ym^ ^ JbrJL ^ 

^ {fU^n M HO yc'O^^ <p^ ^ u^^x^ 

mObr- J^ '/^ Miff ^>CmVi ^yV^. 

An fua- ,<ddr /v-eMU^ ^^/OA^^ton^ ^xj^A^t^A, J^ ^ 

ni^uj- u/^cLi-' ^ c^M^ Ic^ ^^wu u^*w*|^ 

kinds of stories about His past One stoiy he told me touched me so much, ! 
shall never foiget it as Jong as I live. 

When he was 40 years old he wd5 by himself in a cabin out in a forest and 
was accidently shot in the arm by a rifle. In this tragic accident his arm was 
half way blown off, but he didn't panic, he forced himself to get to a train. 
He had to take the train to the ne; rest docter who was about 10 hours away. 

Tile docter said his am was to be amputated or he may die. Grandpa 
Engstrom refused^ He survived the surgery tht strong willed man hv ';;as. 

Grandpa Engstrom is dead now but I still look over in hts old vegtlble 
garden, which is now just weeds and can see him working away. 
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In some readings there will be a group of papers called splitters. 
These papers will tend to be widely distributed in categories. This 
is especially true of the type exemplified on p , 39. 

The student does have some sensitivity to form. He b^ns Math 
a direct quote: **You got me worried, I thought you would never 
come back/* Then the author shifts to a more distant point of 
view: **Then I woke up and my dog wasn't there. I went down 
stairs and told my mom and dad.** If the same point of view had 
been maintained^ the line would have read: "Then» I woke up» and 
you were not there." Finally the author brings the piece to a swift 
conclusion^ increasing the impact of the ending. 

The table leaders must decide how to deal with splitters of this 
kind. The most common solution is to put the paper in the lower 
Iialff reserving the upper half for papers that are mechanically 
respectable and allowing the scoring to shift between a 2 and 3. 
An upper*half paper should not make the reader struggle to under 
stand. What will keep the paper in category two are the anchors 
defining category three. For instance^ the first paper on p. 40^ 
defined as a 3> will keep the paper on p. 39 in category two for 
most readers. 

Now compare the second paper on p. 40 with the first. Both 
papers are about a baseball glove^ but clearly the first one is better 
than the second one. The last paper would be a 2 and the previous 
paper a 3 if the readers decided to keep mechanical lapses in the 
lower half and distribute the upper ranges among those who vary 
in their skills with organization and development. 

The solution for splitters^ then» is to clearly define the difference 
between upper- and lowerhalf papers and to define what should 
be emphasized on either side of the splitter. When a question exists 
about a particular paper» examine the anchors on both sides of tae 
suggested score. 

The Reading 

The head reader runs the reading, and the table leaders monitor 
what happens at each table. Each reader places scored paper& 
directly in front of his or her position^ and the table leaders circulate 
around the table» rereading some of the scor^'^ papers to determine 
whether re 'ders are staying on the target categories. When papers are 
being read £or the second time» the table leaders check to make 
certain that no paper has two scores differing by more than one 
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week.. Siv^eu •««t 

***** * ^'WK «nA 4*W my 



{Transcript] Welloom Baok paU 1 mist you. You got me warede I thotyou 
wod never com back, wear have you ben the past week. You smeU Bad. How 
Bat a Bath, then I wiU feed you, then wee can get in Bed and go to selp. We 
have a Big day to moros. then I wook up and my Dog wa$ent ther. then I 
went down sters and tolb my mom and bab. they sed He we 11 Be Back So I 
went out side. . . . then two weeks wint past, thin I was walking down the 
str^t and I saw my Dog He was hit By a car. Then I tock him hom and I 
Berde hUn in are Back yard. The end 
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points If the difference is more than one points then the table 
leader reads the paper a third time and changes one of the original 
scores^ moving the total of the paper up or down^ Usually no more 
than five percent of the papers need third readings. The table 
leader puts the total of the two scores on all papers^ circling the 
score and undedining it in the center of the first page of the 
sample. 

During the readings the table leader should be watching for 
good sample anchors which can be used for re^choring^ and special 
papers which could be used by the head reader for handling 
special concerns. Table leaders should attempt to keep readers on 
the right anchors. If a reader is obviously scoring too high or too 
low» the table leader should select some test papers and ask the 
reader to score them. Then if the scores are still off» the table 
leader should draw the reader away &om the scoring table and 
confer with the reader on how the misscored paper fits the criteria 
of the boundary categories. In other words» if a reader scores a 
paper as a 2 and the score should be a 3^ ask the reader to compare 
the paper with the 4» then the 3» and finally the 2. Work on com- 
parisons with the boundary categories^ not comparison with other 
papers in the misscored category. If a reader wishes to argue that a 
paper should not be a 2^ then ask whether a boundary paper should 
be a 1 or 3. The reader may find that the paper is different from 
other 2s, but it is certainly not a 1 or 3. 

The reading ^ouki take place in a large room and the scoring 
tables should give the readers plenty of room for putting a sequence 



[Transcript] When I was in 3rd grade I got this baseball glove for my birth, 
day. I donH really remember what the trade name was but I think it was a 
Wilson, I must of had it for tibout 3 1/2 years. And it was the best glove I 
ever had and the best thing at that time. It was broken in real good too. I use 
to oil it all the times that why it was in and good shape. All my friends would 
throw there gloves, butdidnl do that to ofen unless I got real mad. Then one 
day I won^t to get it and it wasn't there. So I looked all over but didnH fined 
it. Then I went out side and saw in on the grass It had been all chewed up by 
the dog So we threw it out and about a week later I got a new one, but I 
didn't like it as much. And it was my favorite glove even. 

(Transcript) lam espestally attache to a old gloveof mine because it was my 
firts glove and 1 love the game of baseball. My grandfather bought the glove 
when i was nine years (^d and Ive had it ever since the reason the glove U 
so valueble to me is because it was the Or^s glove I used in a game when I 
played little baseb?ll and its so small it makes tne laugh because it reminds 
me of old times now the glove still hangs in my closet. 
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of anchor papers face*up in ftorit of them. The head reader should 
stress the following points in the introduction: (I) scoring and 
grading are different^ (2) mixing grade levels does not harm stu- 
dents» (3) talk should be kept to a minimum and questions should 
be directed to table leaders> and (4) read tasU do not think about a 
paper ^':o much» and score your first impression^ making certain it 
fits the anchors in front of you. 

The grading issue is confusing to some teachers. Grading is a 
pedagogical device, sometimes used to encourage students who are 
not doing well. Holisitc scoring is used to get a group profile of 
students^ showing the range of writing skill from top to bottom. 
But the score on an individual piece of writing is not a statement 
of an individual's competency in writing. If oth^ prompts were 
given> if more time were allowed^ if the day were different^ the 
individual's score might change. Because the individual's score 
might change if the prompt were changed^ most districts use more 
than one prompt in the design of a minimum competency vmting 
assessment. And because grades are used for pedagogical purposes* 
and for assessing performance within a particular course without a 
district norm» teachers shoukl not use their grading procedures to 
score papers in a holistic assessment. 

Some teachers worry about mixinggrade levels* fearing that the 
lower grade levels will not receive a fair reading. The fact is that in 
a fourth through twelfth grade readings some fourth graders always 
do as well as some twelfth graders. Grade levels* the readers are 
told^ are separated for study after the reading is over. Putting all 
papers into the same reading ensures that the range of writing 
skill will be expressed in th e distribution of the scores. 

The head reader begins the training of the readers by giving 
them six anchor papers and by asking them to arrange the six in a 
sequence from low to high* using* in this instance* six scoring cate* 
gories. Finally* the readers should mark scores on the papers* I for 
tow and 6 for high. After the readers have finished arranging and 
scoring their papers* the head reader asks them to vote on specific 
papers: "How many gave paper A a 6? How many gave it a 5? 
How many a I» a 2?'' and so on. In the sequence of the voting* 
papers scoring 3 and 4 are saved for last because the middledistinc* 
tion is the hardest. If the anchoring selection has been done with 
care» the votir^ will be clear and certain in its placement of the 
anchors. In a reading on a fourpoint scale at Bret Harte Junior 
High in Oakland* the teachers voted as follows on six papers: 
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How the Teacher Readers 
The Anchor Papets: Voted to Score Papers 

How Leaders Scored Them Score I Score II Score 111 Score IV 



Anchor Score I 


14 


3 


0 


0 


Anciior Score II 


2 


16 


0 


0 


Anchor Score III 


0 


2 


12 


3 


Anchor Score II 


2 


12 


4 


0 


Anchor Score IV 


0 


0 


3 


15 


Anchor Score I 


11 


7 


0 


0 



The vote that signals a problem m this sequence is the last one. 
The anchor 1 got a large number of two votes. This requires that 
in the n'^t sequence^ the anchor for 2 be very strong but clearly 
below the 3 so that some of the seven readers who voted for 2 the 
first time wiU vote for the 1 category when the paper is reconquered 
later The head reader keeps introducing anchors for scoring until 
the trouble spots disappear in the voting. 

At this point in the reading there is very little talk about the 
papers. Experienced teachers have often worked with one another 
for several years and have had numerous discussions on grading 
and literacy. Asking for reasons too early in a reading can easily 
start an argument. The teachers often know each other's position 
already. But scoring papers and building strong categories before- 
hand makes useless argument less likely when it comes time to 
discuss reasons for differences among the categories* Notice also 
that one of the eighteen teacheis in the Bret Harte reading did hot 
vote on two anchors. This is not a problem if the teacher votes on 
later anchors. 

The discussion starts during the reading of the second sequence 
of anchors: "How does category one differ ftom two> two from 
three?** Each table leader is asked to iAke notes on this discus^on^ 
and the head reader can collect these notes and summarize them 
for posting in the reading room. This is especially important for 
the report that comes later. Rubrics should grow out of the 
discussion of readers on how they established the categoriesp 

The head reader should ask the readers to put the first score on 
the back of the paper in the center of the page. The second score 
should be put on the ftont of the paper in the upp^ right-hand 
comer. The final total score^ prepared by the table leaders and 
their assigned assistants^ will be put at the ftont-center of the first 
page» circled^ and underlined twice. If the table leaders find that 
an individual reader continues to give odd scores* the table leaders 
should assign that reader to the head reader for special duties^ such 
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as collecting papers and counting them. In the transition from first 
to second readings^ the head reader must circulate the papers cor- 
rectly so that different readers do the second reading. 

As the final papers axe being scored, the head reader should begin 
to arrange the room for counting papers and asking readers to total 
their categories. When two readings are added together^ the six- 
point scale produces an eleven-point scale beginning with two. The 
readers should, if possible^ know the totals before they leave the 
reading. A recent reading conducted in the San Mateo Elementary 
District produced the following totals: 

Scores: 12 11 10 9 8 7 6 5 4 3 2 
Totals; S5 109 251 306 567 534 787 344 382 51 52 

What is interesting about these totals is that the distribution 
approximates a bell-shaped curve. The Central Limit Theorem 
in statistics tells us than any variable which is itself the sum of 
other random variables will have a normal distribution. The ap- 
proximation of the bell-shaped curve is one way of checking 
whether the reading did an adequate job of distributing the range 
of writing ability across the six scoring categories. What is also 
encouraging about the distribution shown above is that the num- 
bers scoring 12 and 11 exceed the numbers scoring 3 and 2. Some- 
times one worries that teachers will not let any students attain the 
top categories. If the sample is large enough^ if the sample is from 
mixed classes, if the anchoring has been done correctly, and if the 
readers stick to the anchors^ the distribution should be approxi- 
mately belUshaped. 

To provide space for all the activities that must take place 
during a reading, the head reader should arrange the layout of the 
reading room approximately as shown on p. 45. 



The Summary Checklist 

1. Select table leaders and estimate number of teachers and days 
for the reading. 

2. Table leaders select anchors from sample. 

3. Table leaders identify traits of scoring categories. 

4. Table leaders score papers to test reliability of anchors. 

5. Table leaders identify papers that focus on a particular prob* 
lem, especially the relative weights of fluency and coherence. 
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6. Begin reading by having readers score a set of anchors^ one 
for each scoring category, 

7. Have readers score a second set of anchors with more papers 
in the middle categories. 

8. Have readers score special anchors which focus on any problem 
trends noted in the voting, 

9- Have papers read twice by different readers and a third time 

if the two scores differ by more than one point. 
10, If the writing samples represent the normal school population^ 
not just those who faOed the previous test» then estimate how 
well the anchors distributed the scores by seeing whether the 
distribution of scores approximates a beU*shaped curve. 



The Report 



The report prepared on the reading can include four parts: (1) the 
rubric describing the six categories^ including diagnostic infonna^ 
tion; (2) the chart$ showing the distribution ;(3) PK>£icien<^ reports; 
and (4) foature analysis. 

The preparation of the rubric begins with each table leader 
taking notes on comments that teachers provide for each scoring 
category. Later the scoring categories are analyzed for features that 
deGne the categories. Sometimes the rubric is a careful breakdown 
of the prompt and directions^ as in ttie rubric for the essay exami^ 
nation in California State Colleges (White, 1973): 

Many obser^rs of our so^^ daim that modem man, immetsed 
in materialism, is **owned by his objects.*" Yet many of us have 
objects that we treasure not just for their material value but fora 
variety of other reasons* Assif^nment: Describe one or moreobjects 
which are important to you. Explain what values they represent 
and comment on those vahies. 

The fallowing directions for scoring were distributed to all 
readers engaged in the grading of ijuestion 1 : 

The student should be rewarded for what he does well in 
response to the ijuestion. Here the student is asked to desaibe 
one or more objects important to him. He is further asKed to 
explain what values they represent and to comment on those 
values. He is told to think about the question and to plan his 
response. 

Note that the question does not demand that the object or 
objects be unusual ones. 

Essays which mbinterpret "objects" as "objectives " and which 
deal mainly with generalized at»stractions (life, God), should 
be read sympathetically, but they should ordinarily not receive 
above 2, since they fafi to understand and properly respond to 
the question. 

Possible Scores: 

6 A superior response will not just name one or more objects but 
describe them in some detail, and it will not just identify the 
values represented but oxptain and comment on them, their 
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nature and Iheir soum. A superior paper wiU be literate and 
orderly. 

5^ These scores witt be useful for a well-handled paper which is 
deficient in one or two characteristics of the superior response, 
i.e.^ in description of the object or objects and In explanation 
of the values represented, but which is otherwise competently 
written. 

V2 These scores will be useful for the following kinds of papers: 
-Hhosc in which only one part of the two-part question Is 
addressed; 

-Hhose in which the representatK-eness of speciHc objects is 
ignored; 

-^ose which treat the subject in superftctal or stereotyped 
fashion; 

-Hhose in which the writing exhibits several weaknesses^ In 
wording and other respects. 

1 This score is to be used for papers which are lacking In focus 
and substance, depart from the assigned topic^ and/ir exhibit 
serious writing faults. 

* Non^responsc papers and papers which do not fall into the 
foregoing categories^ extensively argue with the question, or 
are otherwise idiosyncratic^ should immediately be brought to 
the attention of the table reader and the question leader. 

On the favoriteK>bject topic» the California assessment of the 
vmting of high school seniors used a somevtrhat different approach 
for the rubric and the holistic scoring. Firsts the scoring procedures 
asked the readers to give each paper tvtro holistic scores^ one for 
the ^'writer^s overall sense of composition** and another for the 
"vmter*s overall competencies vsrith written English.** Then each 
reader was given the following rubrics^ one for composition and 
one for competency: 

Score for Composithn 

First, the reader assigns an even number score according to the 
fbDowing descriptions: 

8 The writer identifies an object, supplies descriptive details^ and 
provides a strong accounting for his or her attachment to the 
object either through lively anecdote or an insightful discussion 
of values (usually more than one value). The paper has almost 
no Irrelevant statements and has an easily observable coherence 
or plan. The writer wastes no space on talking about how hard 
it Is to think of something to write about, nor does he or she 
begin slmpllstically with a formula, such as, ^The object I am 
going to write about is* . . 

6 The writer gives some descriptive detail about the object and 
at least one examined reason Tor his or her attachment to the 
object The paper has no serious Incoherence and little in the 
way of ine[e\'ant or drgressive statements. 
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4 The writer merely names the object and says a bare minimum 
about his or her reasons for having the attachment to the 
object; or the writer describes an object without stating or 
implying much about the personal attachment. In some cases 
the .writer may confuse the issue talking about drawbacks 
or disadvantages to the possession (for example^ wanting to 
sell it and g^t a better one). The paper may be somewhat 
incoherent or contain clearly irrelevant statements or issuer. 

2 The writer does not seem to have understood what he or she 
has been asked to do. For example^ the paper may simply 
volunteer a topic which has not been called for. The paper will 
usually contain obvious irrelevancies or be so halting and brief 
that it communicates httle Infonnaiion to the reader. 

ScoTO for Competency in English 

The reader judges the writer's overall skills in punctuation^ dic- 
tion and usage, and sentence sense. Scores are assigned according 
to the following broad descriptions: 

5 Outstandingly varied, mature sentences, with strong diction 
and almost no errors; notable in use of subordination and free 
of modifiers. 

A Above-average sentence sense; few errors In usage or punctua- 
tion (and none of them major); perhaps some hint of a sense 
of style. 

3 About average for the papers scored; many short sentences, 
perhaps with some comma faults and an occasional fragment; 
but overall^ a suggestion that the student has a fair grasp of 
sentence sense. 

2 A cluster of errors in grammar^ usage, and punctuation, or a 
paper which hasa dcad.|evel simplicity and monotony of style; 
practically no sentence without an error of some kind; but in 
spite of these problems, generally readable. 

1 A paper so badly marred with problems that the reader can 
hardly follow the ideas presented, either because of stumbling 
syntajc (mijced structures and the like) or because of word 
choices and such egregious form problems (including spelling) 
that the reader finds it difficult to be sympathetic with the 
writer. 

ATofe: Matters of spelling are intentionally nor specified in the 
descriptions of papers rated 5, ^, 3, and 2. The reader should ig- 
nore spelling except for the 1 paper. This categoiy allows for the 
paper with so many misspelled words that the reader muifit actually 
struggle to folbw the thread of thought. Such a papei suggests 
a degree of near illiteracy. 

This procedure can work quite will when there is plenty of time 
for training and when experienced readers and table leaders are 
available. But in most readings^ the best approach is a careful selec* 
tion of anchors^ a development of a rubric from the anchors^ and 
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then scoring with one holistic score^ providing time for reanchohng 
and consideration of special problems. 

Changes in grade level can sometimes influence the rubiic. For 
example^ the foUowin^; prompt and rubiics were used in a third 
grade reading (Martin, 1976-77): 

Writing Sample Topic 
Grade Three 

[Students were asked to study a photograph of an elephant doipg 
a headstand with the apparent **help" of a small boy supporting 
one hind leg.) 

Students: Look carefully at the picture. Then think about what 
might have happened before the [^cture was taken» what is going 
on ii) the picture^ what will possibly happen next» and who else 
might be involved who is not in the picture. Then write a stoiy 
about your ideas. The {dea is to write about what the picture 
means to you; use a little time for planning before you start to 
write. 

If you have time before the papers are collectcd^pteasecheck back 
over your ^^tory for punctuation^ capital letters^ f^ood sentences^ 
and spelling. 

You may take 30 minutes^ but no papers should be turned in to 
the teacher before 15 minutes^ so take the time to make a good 
story. If you want to use a now word whose spelling you are unsure 
of, use it any\vay. 

Hand\vriting tends to become less important as the students get 
older. In the Jefferson County^ Colorado^ rubric (Jefferson County 
School District^ 1976; see facing page)^ the t-eachers included 
handwiting as a critical feature in every scoring category through 
the ninth giade. By the eleventh grade^ handwiting was listed as 
a critical feature only in the lowest category and then Uie only 
characteristics listed were negative. 

The rubric provides some general information on how distinc- 
tions are made among the categories^ and the rubric is always 
distributed with samples of the anchors so that the public or other 
teachers can check their own intuitions by reading papers. A rubric 
without sample anchors is to a targe degree uninformative. In 
addition to the rubric^ teachers May decide to include some 
papers that illustrate specific problems that need correction. The 
diagnosis with the sample gives some help to those teachers who 
are uncertain of how to help students who do not do well in the 
assessment. 
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The next six papers illustrate how diagnosis is published in the 
assessment report. The first two samples show the problem of stu- 
dents who need fluency: 

^ ii2^i 1009 -^toz 

v\t\di T- put tW(u \\ rtiNj 

hMf ror. 

My f<^\Jov\X obJer 

These two students need practice in writing in journals every day» 
filling paper with notes on what they have ob^rved» writing dia- 
logues back and forth with neighbors (an exchange of written notes 
with a student nearby improvisations— whatever finger exercises 
help get the flow of thought going so that the student has some 
fluency on the written page. Teaching these two students letter 
formation and run-on sentences will only inhibit an already deeply 
inhibited inclination to write. The two students ^vhose papers 
appear on pages 54 and 55 have a different problem. They have 
fluency but very little coherence. In fact* "Me and Lego" is almost 
incomprehensible. These t^vo students need audience responses in 
writing groups^ and sentence combining exercises so that they learn 
to combine ideas in some way other than "and.** They need prac- 
tice in revising papers after work in student editing groups> hearing 
another student read their papers aloud. They might even profit 



{Ttanscnpt} My Nanna gave me a two dollar bill and I have 10 dollar and 
[ put them It my mirror. 

My Favont objet is a stero. My stero Is black and brown. I ded bay weth 
my money in Chismas. 
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fxom a few exercises on editing principles which they seem to ignore 
or do not know. But sentence combining and audience awareness 
are the primary needs at the moment. 

The following student seems in need of attention to both fluency 
and coherence: 

da-i>s> CLr\J ii'JU^ yun^trt^ ck Zcx^u-tyu ^^UVf^ 



This student needs practice with sentence combining and editing 
of spellings This is the type of student who has developed some 
fluency; he has a point he wants to make and does not appear to 
hesitate to do it in writing. But the student m^t be overwhelmed 
by being told of all the things that are wrong with the essay. It 
appears that the student could easily leam sentence combining 
skills, and in the meantime the editing of the spelling might be 
done by an assigned pBjiiner. Later» after sentence combining skills 
are reflected in the writingj the student could be given two words 
per week to leam, b^inning with "whud" and "mone/* later 
"cuntree** and **lowyer.** I am uncertain whether reading aloud 
would help. The sound seems to be there, as if the student could 
actually hear the words. 

The student who wrote **My favodte object'* on p. 56 is quite a 
different case. This student already thinks of himself as a writer. 
He has an idea for tlie interesting detail (''blowing the dust off" 
the comic book) and the dramatic implication ("remember it*s 

[Transcript] I wish I had ^ motersikle I Khud ride all ove the cuntree side 
and if 1 had a mellon doUors I whud have a jet airplane I wude visit all my 
reletlves and Sae howoa are you doing a am doing ftnd and then 1 wood 
get a lowyer and let hem investes the rest of my monei and tet hem tnppte 
and then I wud by mis Spikes a big house and then by my five daughters 
gifts and wee cud live one easy streek. 
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[Transcript) Me and Lego/0 Once upon a time 1 think it is a good thing to 
play with. I think when you have nothing to do. And I think that when you 
have fun. And you can do stufT. when you have something ta do. I go when I 
have nothing to do and I play whith my fiends with it. The thing abet my 
lego I thing St is a good thing to play with when you have noth to do. And I 
have a lot of with my lego. Evey time I paty with my lego and I made a car 
and a truck. 1 think abet my lego it is a good thing to play with and I thing it 
shund be a good thing to play with. And it col be a good thng and I can do 
stoff with my lego. And it shud hav a good stuff to do with my lego We can 
makealotof stuff of lego. I thing it is a good thing to do. 
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[Transcript] I remember one day I was in the store and this girl came up to 
me and said. "Hi! Klnetha." I looked at her veiy particuly, you know, like 
who are you? she satdt "Jackie from Vallejo where you use to live, I said 
"oh, and then from ther« on I fergot about my other friends because, I 
r«member her and I haven seen her in along time. And we talked about our 
old days when we were veiy little. So we went to Santa Cruz and I seen Carol 
Gamison^ She said "Hi to me and I looked twice and then I remember her 
from School at King, I said oh I Carol, as if I never even knew her, and we go 
to school together eveiyday, I mean, I completely forgot all about her, cause 
my good friend and I been having such a good time eveiy since we remember 
each other that day. It's been very exciting being with her and she must 
go backy to Vallejo wher« she lives now. Bye Jackie!! Call me sometimes. 
She 0,K, 
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magic**)- But he has a few mechanical lapses^ and in some parts he 
might have aimed for more elegance. Nevertheless, he is good. 
What this student needs is an audience— having his paper posted on 
the bulletin board in the school hallway* seeing his writing printed 
in the school newspaper or reproduced on ditto paper and distrib- 
uted to others* having the teacher read his writing to the class, 
having other students read and comment on his writing in editing 
groups, A keen sense of audience and an awareness of how writing 
is judged is probably all this student needs to improve his already 
good work. 

The preceding six students are useful cases for demonstrating 
how diagnosis can work with individual students. The report could 
also include group information on the distribution of papars* the 
features counted in various categories, and the stgniBcant differ* 
ences^ if any* between pre- and post-tests. The charts on pages 
63-65 show pre-test and post-test results in a writing assessment. 

These charts were prepared for the assessment of writing samples 
in grades four through twelve in Oakland* California* with all of 
the papers being read at the same time on a scoring range of 1 
through 6, Because the two scores were addiid, the reported range 
is 2 through 12, Note that in grade four* the papers primarily range 
&om 2 through 9* and in grade six and above the range is 2 tiurough 
12, Furthermore* notice that from seventh through twelfth grade 



[Transcript] My favorite object/I have a magic comic book, every time I open 
it, it tums me into tiie super iiero on tiiat page! One day it gave me tiie 
powers of Spiderboy, anotiier day I was Batboy, 

Most people asic me wiiere I got tiiis comic iioolc but I never teit tliem 
but youVe luclcy, Im going to teii you wiiere 1 got my boolc and iiow 1 
got it 

One day wiien 1 was in San F^cisca I wandered in to tiiis old comic 
boolc store Tiiere was nobody In there but an old man, I browsed for a 
whiie till a little comic boolc caught my eye, 1 picked up, after blowing 
the dust off it I read Super Hero Digest,*' I bought the Book for a dime and 
as 1 was leaving the old man called to me ''remember, it's magic!** 



ERIC 



58 



The Report 



the number of scores in the top ranges increases. One can conclude 
that as the students get older, the group averages on the writing 
assessment increase. In addition, the solid line and the dotted line 
show the differences in writing scores from September to May< 
After the middle^ the dotted line should be ahove the solid line 
if papers are getting better. 

Many writing assessments are given to satisfy state requirements 
for proficiency standards, a test to determine whether a student 
has satisfied the minimum requirements for graduation. Teachers 
who participate ui a writing assessment for purposes of minimum 
competency definitions should exercise great care in the process 
for selecting the point at which students pass or fail. Many districts 
have found that the best procedure is to complete the reading as 
outlined in the previous pages, tabulate the results, and then send 
the group results and anchor papers to the district proficiency 
committee and ask the committee to decide where the cut-off 
point should be* This procedure prevents a situation in which the 
readers are charged with being *'too easy'* or ^''too hard*' in decid- 
ing the cut-off. A single group of readers should not be charged 
with the responsibility of deciding how many students are judged 
incompetent. 

Second, no single writing sample should be used as the basis for 
deciding an individual's competency in writing. Twenty samples 
would be better. But districts cannot afford to collect and score 
twenty samples so some compromise will be necessary. In Oakland, 
California, the students took three tests for writing skills-two 
writing samples and one mechanics test. The student was required 
to pass any two of the three. The proficiency procedure should 
also provide a chance for the student to appeal the competency 
decision, providing several opportunities for the student to write 
the samples and have them scored. Using the same anchors coukj 
help keep the scoring range relatively intact. 

Feature analysis is also included in some reports on writing 
assessment. The analysis displayed in Tables 5 and 6 of the National 
Assessment (National Institute of Education, 1975) is reproduced 
on pages 66 and 67. 

Feature analysis is best done after the scoring is complete. 
Simply select a random sample of papers from each scoring cate- 
gory and count the features that concern the school district. The 
results of the count can provide baseline data for discussing results 
in other readings. Districts often staart with a list of features like 
those in Table 6 as the scoring guides for assessment. This list is 
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usually prepared ftom a list of errors found in the district curricu- 
lum guide. 

But the error count can be very misleading. An excellent paper 
can have a spelling error Some counts are unequal. For instance^ 
examine the following: 

1* The people is running down the street. 

2. The people^ their clothes on fire and anguish in their eyes> 
is running down the street. 

3^ Charles Jones» who owns the car» is here. 

4. The man> who owns the car> is here* 

The subject^verb agreement in the second sentence is probably 
less serious than the subjectwerb agreement in the first sentence. 
Sometimes students who are adding embedded modifiers for the 
first time» as in the second sentence^ lose track of the subject-verb 
agreement relatioa'^ip. The error seems more serious in thi^ 'irst 
sentence because the subject and verb are next to each other If 
there Is a comma error in the third md fourth sentences^ is the 
error counted as one or two? If it is counted as two» then the single 
problem of setting off nonrestrictive clauses is being given double 
weight in relation to other problems* The point is that error count* 
ing is not as simple as it sometimes seems. 

However^ the easiest feature analysis is the type that counts 
errors and other surface conventions. The most difficult feature 
analysis is the type that attempts to examine discourse issues that 
are below the surface. In other words^ one must use surface features 
as a reflection cf a deeper issue. One example would be the exami- 
nation of oral language traits and written language traits in the 
anchors for various scoring categories. The issue is that children 
usually develop their oral language in dialogues before they are 
asked to give formal speeches or monologues^ but in writing^ as 
Roger Shuy has indicated » students are usually asked to produce 
monologues without adequate experience at some dialogue level 
in writing. 

The writing samples^ then> should show the students attempting 
to keep the writing close to an oral dialogue or else moving easily 
into a written monologue. I examined 120 papers^ 20 at each 
scoring levels from a reading in Oakland^ California^ in order to 
determine whether there was any pattern of oral and written lan- 
guage at the various scoring levels (Myers» 1978). The oral language 
traits were called information processii^> and the written language 
traits were called text packaging: 
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Information Processing 

1, Connectors: and» so» 
then» well^ now 



TBxt Packaging 



1- Relatom hnty either/or» 

whilet wheiif since^ if* who^ 
which* therefore* however* 
for example* and other 
subordinaiors 



2. Hedges: sort of* kind 
of* maybe* probably* 
and so forth 



2> Qualifiers: ina sense* perhaps* 
rather* somewhat* it is 
possible that* relatively* 



and so forth 

3> Leaps: a lot* just* espe^ 3> Generaiizers: in general* in 



These tracts and others were counted and divided by the number 
of words in the sample* producing an index for information 
processing (or oral language traits) and text packaging (or written 
language traits). The other traits counted included the number of 
clauses with five words or less and twenty words or longer-the 
assumption being that oral language patterns were governed by 
limited turns in dialogue* and that written languagie patterns were 
governed by extended turns in monologues. The headings ''infor- 
mation processing** and ''text packaging" were given to the two 
types of language on the assumption that students use the oral 
dialogue pattern to generate information and use the written 
monologue patterns to package texts. The graph on the facing 
page diows the pattern of information processing and text pack- 
aging in tlie various score categories. 

The graph shows that between Levels I and 11 papers and Level 
111 papers* the distance between text packaging and information 
processing increases* suggesting an increase in fluency without a 
consequent increase in those traits which improve the organizational 
coherence of the papers* At Level IV* however* the text packaging 
index is larger than the information processing index* and the 
papers in the VI category show a decrease in information processing 
traits and an increase in text packaging traits. 



cially*very* sure* a 

real * all tlie time* 

only* plenty* and 
so forth 



the most part* typically* 
and so forth 



essence* essentially* it is 



true that* in deed* certainly* 
without doubt* largely* for 



4- Use of you (close 
audience) 



4- No you (distant audience) 
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This finding from the wnriting assessment m Oakland, Califomta^ 
has been shared with teachers m insorvice programs, and they have 
been asked to speculate about wliat the results might mean in 
terms of program change. Writing teachers ate generally agreed 
that current time writing programs do not make explicit the differ- 
ence between ora] dialogues and written monologues^ and that if 
these differences were made clear to students^ students would 
show improvement in their scores. One way to make the differences 
clear is to ask students to translate oral dialogue language ("sorta/' 
**kinda*') to written monologue language ("it is possible that/' "in 
a sense''). Tlie approach is still being explored. Tlie point is that 
analysis of papers can help teachers explore some of the more 
fundamental issues hi the teaching of wrjthig. 

The writing assessment and its subsequent report can give the 
teaching of writing great significance in adlstrict or school program. 
In addition^ the writing assessment procedures can be used by a 
teacher to score papers in the classroom- The great benefit to 
participants is the awareness of the range of writing skills in aclass^ 
schooU or district and a recognition that when teachers elect 
prototypes of competency, they can usually agree on issues of 
merit. 
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TABLE ^- Average Poor and Good Essays Written by 
IV-Year^Olds, CounU 





1989 


1974 


Change 




Poor 


Good 


Poor 


Good 


Poor 


Good 


Avtnsiv holistic 
score 




7,2 


2,6 


7,2 


^2* 


*t 


Av«^c number 


74,7 


176,4 


67,3 


218,6 


'7,4 


42,2* 


Av«raC« mimbeT 


5,2 


tl,2 


4,4 


14,3 


'0,8 


3,1* 


Aver«Re number 
ol Pmr^tTAPhsyestay 


t,4 


2,1 


1,1 


2,2 


'0,3 


0,1 


Av«Tafte number 

ot Punctuation cni^rK^ 


7,6 


19,4 


7,2 


26,9 


'0,4 


7,5' 


letters/ word 


4,0 


4,3 


3,9 


4,3 


'0,1 


*t 


AveT&K« number of 
words /sentence 


16,6 


16,6 


J 8,1 


16,3 


1,0 


'0,3 


Aveiaec number of 
words/Pmra£raPh 


6t,9 


115.3 


63,6 


135,3 


1,7 


20,0 


Averasc number of 
sentences fp4r«&rap h 


4,3 


7,1 


4,2 


8,9 


^0,1 


1,8 



^Differences lhat are statisticatty significattl are indicated by ostertsfts, 
Wus signs equal rounded numbers less than 0,05 and rounded percents 
less than 0,$, 
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TABLE 6. Avoiagc Pooi and Good Essays Written by IT-Year-OWs* Sentences 
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(ences/ 
Essay 


Average 
Numbef 
oi Stn* 
t^nees/ 
Essay 


Average 
Percent 
of SeA' 
tenoe$/ 
Essay 


Sim Pie sentences 


2,8 


4S^ 


£,1 


47% 


2,1 


40% 


7,3 


499& 


-0,7 


'Z% 


2,2* 


2% 


Compound sentenees 


0,3 


5 


0.6 


6 


0.2 


6 


0,7 




'0,1 


1 


0,1 


*1 


Complex senUnCei 


K2 


:i5 


4,1 


37 


1,0 


25 


5.2 


38 


-0.2 




i.r 


1 


RuH'Ons 


0.6 


14 


0,3 


3 


0,1 


20 


0,4 


4 


0,1 


6 


0.1 


1 


PiaSmcnti (inoorreeO 


0,4 


1 


1,0 


6 


0,1 


9 


0,5 


3 


J 


2 


0,5 


'3 


Sentences with P biases 


3,2 


63 


8,7 


81 


2,6 


55 


11,0 


7» 


0,6 


*8 


2,3* 


'3 


SfmPte s<:ntences 
with Phia^es 


2,0 


36 


4,5 


41 


1,6 


29 


5.9 


40 


-0,4 


'7 


1.4* 


'1 


Complex ^nt«nees 
with Phiafes 


1,0 


21 


3,7 


34 


0,9 


21 


4.5 


34 


-0,1 




0,8 




Awkw^d *enteno^*j 


1,3 


24 


1,0 


9 


1^ 


26 


1,6 


12 


-0,3 


2 


0,6* 


3 



*Otffcrcncct that am ttatitticalty ttarttftcant <ire inditQted bV ^ttcritkt, 

r/'iuf ^tgn CQu<ttt roundtd nittnbcrt t^it thon O.OS and bounded percentt tct$ than 0.5, 
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